
\chapter{Overview}



\section{What is Model Validation?}

Key to ensuring the quality of the software is ongoing validation testing of the model. Validation typically involves comparing model simulations with experimental measurements. To say that the Consolidated Model of Fire and Smoke Transport (CFAST) is ``validated'' means that the model has been shown to be of a given level of accuracy for a given range of parameters for a given type of fire scenario. Although the CFAST developers periodically perform validation studies, it is ultimately the end user of the model who decides if the model is adequate for the job at hand. Thus, this Guide does not and cannot be considered comprehensive for every possible modeling scenario.

Although there are various definitions of model validation, for example those contained in ASTM E1355~\cite{CFAST:ASTM:E1355}, most define it as the process of determining how well the mathematical model predicts the actual physical phenomena of interest. Validation typically involves (1) comparing model predictions with experimental measurements, (2) quantifying the differences in light of uncertainties in both the measurements and the model inputs, and (3) deciding if the model is appropriate for the given application. This Guide only does (1) and (2). Number (3) is the responsibility of the model user.

The following sections discuss key issues that you must consider when deciding whether or not CFAST has been validated. It depends on (a) the scenarios of interest, (b) the predicted quantities, and (c) the desired level of accuracy. Keep in mind that CFAST can be used to model most any compartment fire scenario and predict quantities of interest, but the prediction may not be accurate because of limitations in the description of the fire physics, and also because of limited information about the fuels, geometry, and so on.


\section{Blind, Specified, and Open Validation Experiments}

ASTM E1355~\cite{CFAST:ASTM:E1355} describes three basic types of validation calculations -- {\em Blind}, {\em Specified}, and
{\em Open}.
\begin{description}
\item [Blind Calculation:] The model user is provided with a basic description of the scenario to be modeled. For this application, the problem description is not exact; the model user is responsible for developing appropriate model inputs from the problem description, including additional details of the geometry, material properties, and fire description, as appropriate. Additional details necessary to simulate the scenario with a specific model are left to the judgement of the model user. In addition to illustrating the comparability of models in actual end-use conditions, this will test the ability of those who use the model to develop appropriate input data for the models.
\item [Specified Calculation:] The model user is provided with a complete detailed description of model inputs, including geometry, material properties, and fire description. As a follow-on to the blind calculation, this test provides a more careful comparison of the underlying physics in the models with a more completely specified scenario.
\item [Open Calculation:] The model user is provided with the most complete information about the scenario, including geometry, material properties, fire description, and the results of experimental tests or benchmark model runs which were used in the evaluation of the blind or specified calculations of the scenario. Deficiencies in available input (used for the blind calculation) should become most apparent with comparison of the open and blind calculation.
\end{description}
The calculations presented in this Guide all fall into the {\em Open} category. There are several reasons for this, the first being the most practical:
\begin{itemize}
\item All of the calculations presented in this Guide are routinely rerun. The fact that the experiments have already been performed and the results are known qualify these calculations as {\em Open}.
\item Some of the calculations described in this Guide did originally fall into the {\em Specified} category because they were first performed before the experiments were conducted. However, in almost every case, the experiment was not conducted exactly as specified, and the calculation results were not particularly useful in determining the accuracy of the model. \item None of the calculations were truly {\em Blind}, even those performed prior to the experiments. The purpose of a {\em Blind} calculation is to assess the degree to which the choice of input parameters affects the outcome. However, in such cases it is impossible to discern the uncertainty associated from the choice of input parameters from that associated with the model itself. The primary purpose of this Guide is to quantify the uncertainty of the model itself, in which case {\em Blind} calculations are of little value.
\end{itemize}


\section{How to Use this Guide}

This guide presents a compilation of past and present validation exercises for the CFAST model.  The structure of the report is as follows:

\begin{itemize}
\item Chapter 2 includes a review of published literature by the National Institute of Standards and Technology (NIST) and others on verification, validation, and sensitivity analysis of earlier versions of CFAST.
\item Chapter 3 discusses verification of the model.  A range of analytical calculations are compared with equivalent model predictions. These are intended to test the correctness of specific model calculations.
\item Chapter 4 provides a summary of the experiments used in the current evaluation for the CFAST model.
\item Chapters 5 through 10 quantifies the comparison of CFAST predictions with experimental measurements for a number of important model results.
\item Chapter 11 summarizes the results of the validation study.
\item Appendix A describes calculations used to estimate layer temperatures and interface height from individual temperature measurements in the experiments.
\item Appendix B summarizes the results of verification testing detailed in Chapter 3.
\item Appendix C includes graphs of experimental results and model predictions for all comparisons included in the validation study.
\end{itemize}


As CFAST continues to develop, it will expand to include new experimental measurements of newly modeled physical phenomena. With each change in CFAST, the validation and verification tests are all redone to ensure that changes to the model are consistent with experimental measurements and the overall accuracy of the model is maintained by comparing the results between the old and new versions of the model. If you are embarking on a validation study, you might want to consider the following steps:
\begin{enumerate}
\item Survey Chapter~\ref{Survey_Chapter} to learn about past efforts by others to validate the model for applications similar to yours. Keep in mind that most of the referenced validation exercises have been performed with older versions of CFAST, and you may want to obtain the experimental data and the old CFAST input files and redo the simulations with the version of CFAST that you plan to use.
\item Identify in Chapter~\ref{Experiment_Chapter} experimental data sets appropriate for your application. In particular, the summary of the experiments found in table~\ref{Test_Parameters} contains a table listing various non-dimensional quantities that characterize the parameters of the experiments. For example, the global equivalence ratio of a compartment fire experiment indicates the degree to which the fire was over or under-ventilated. To say that the results of a given experiment are relevant to your scenario, you need to demonstrate that its parameters ``fit'' within the parameter space outlined in Table~\ref{Test_Parameters}.
\item Search the Table of Contents to find comparisons of CFAST version 7 simulations with specific parameters from relevant experiments. For a given experiment, there may be numerous measurements of quantities like the gas temperature, heat flux, and so on. It is a challenge to sort out all the plots and graphs of all the different quantities and come to some general conclusion. For this reason, this Guide is organized by output quantity, not by individual experiment or fire scenario. In this way, it is possible to assess, over a range of different experiments and scenarios, the performance of the model in predicting a given quantity. Overall trends and biases become much more clear when the data is organized this way.
\end{enumerate}
The experimental data sets and CFAST input files described in this Guide are all managed via the on-line project archiving system. You might want to re-run examples of interest to better understand how the calculations were designed, and how changes in the various parameters might affect the results. This is known as a {\em sensitivity study}, and it is difficult to document all the parameter variations of the calculations described in this report. Thus, it is a good idea to determine which of the input parameters are particularly important for your application.








